class: inverse, center, middle # The performance of small-area mortality estimation models ## A simulation study .large[Benjamin Schlüter] .large[Bruno Masquelier] <br/> .large[Dutch Demography Day| 16 Nov 2022] <img src="data:image/png;base64,#logo_UCL2.png" width="20%" style="display: block; margin: auto;" /> <img src="data:image/png;base64,#logo_DEMO.jpg" width="20%" style="display: block; margin: auto;" /> --- # Context Age-specific mortality rates (= mortality age schedule): input for demographic indicators * Standardized mortality rates * Life expectancy at birth, `\(e^0\)` * Lifespan variation Why do we need accurate mortality estimates by age and subnational areas? * Document health inequalities * Identify the determining risk factors of mortality * Guide resource allocation * Assess policy changes at local level * Target areas most in need ??? * __Within__ a country... -- .center[
.highlight[Small population = unreliable mortality measurements]
] --- # Stochasticity in death counts .pull-left[ <div class="figure" style="text-align: center"> <img src="data:image/png;base64,#index_files/figure-html/fig_stoch_fr-1.png" alt="France 2007, female" width="80%" /> <p class="caption">France 2007, female</p> </div> ] -- .pull-right[ <div class="figure" style="text-align: center"> <img src="data:image/png;base64,#index_files/figure-html/fig_stoch_be-1.png" alt="Belgium 2007, female" width="80%" /> <p class="caption">Belgium 2007, female</p> </div> ] -- .pull-left[ <div class="figure" style="text-align: center"> <img src="data:image/png;base64,#index_files/figure-html/fig_stoch_dist-1.png" alt="Walloon region 2007, female" width="80%" /> <p class="caption">Walloon region 2007, female</p> </div> ] -- .pull-right[ <div class="figure" style="text-align: center"> <img src="data:image/png;base64,#index_files/figure-html/fig_stoch_muni-1.png" alt="Brussels region 2007, female" width="80%" /> <p class="caption">Brussels region 2007, female</p> </div> ] -- .center[ .highlight[Variability] & .highlight[Zeros] ] --- # Selection criteria Several models have been proposed to overcome these difficulties, mostly Bayesian hierarchical models (BHM) <br/> * .highlight[Endogeneity concerns] * Models without area-level covariates * .highlight[Life tables] * Models estimating complete mortality age schedules * .highlight[Automation] * Models estimating at all subnational population sizes --- <table class="table" style="width: auto !important; margin-left: auto; margin-right: auto;"> <thead> <tr> <th style="empty-cells: hide;border-bottom:hidden;" colspan="1"></th> <th style="border-bottom:hidden;padding-bottom:0; padding-left:3px;padding-right:3px;text-align: center; " colspan="2"><div style="border-bottom: 1px solid #ddd; padding-bottom: 5px; ">Demographic regularity</div></th> <th style="border-bottom:hidden;padding-bottom:0; padding-left:3px;padding-right:3px;text-align: center; " colspan="4"><div style="border-bottom: 1px solid #ddd; padding-bottom: 5px; ">Pooling</div></th> <th style="empty-cells: hide;border-bottom:hidden;" colspan="1"></th> </tr> <tr> <th style="text-align:center;"> Model name </th> <th style="text-align:center;"> Functional form </th> <th style="text-align:center;"> Penalize deviations </th> <th style="text-align:center;"> Space </th> <th style="text-align:center;"> Age </th> <th style="text-align:center;"> Time </th> <th style="text-align:center;"> Sex </th> <th style="text-align:center;"> References </th> </tr> </thead> <tbody> <tr grouplength="2"><td colspan="8" style="border-bottom: 1px solid;"><strong>Non-BHM</strong></td></tr> <tr> <td style="text-align:center;padding-left: 2em;" indentlevel="1"> D-splines </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> X </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> Schmertmann (2021) </td> </tr> <tr> <td style="text-align:center;padding-left: 2em;" indentlevel="1"> TOPALS </td> <td style="text-align:center;"> X </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> Gonzaga & Schmertmann (2016) </td> </tr> <tr grouplength="4"><td colspan="8" style="border-bottom: 1px solid;"><strong>BHM</strong></td></tr> <tr> <td style="text-align:center;padding-left: 2em;" indentlevel="1"> BHM TOPALS </td> <td style="text-align:center;"> X </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> X </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> X </td> <td style="text-align:center;"> Rau & Schmertmann (2020) </td> </tr> <tr> <td style="text-align:center;padding-left: 2em;" indentlevel="1"> BHM SVD </td> <td style="text-align:center;"> X </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> X </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> X </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> Alexander et al. (2017) </td> </tr> <tr> <td style="text-align:center;padding-left: 2em;" indentlevel="1"> BHM </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> X </td> <td style="text-align:center;"> X </td> <td style="text-align:center;"> X </td> <td style="text-align:center;"> Zhang et al. (2019) </td> </tr> <tr> <td style="text-align:center;padding-left: 2em;" indentlevel="1"> BREM </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> X </td> <td style="text-align:center;"> X </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> X </td> <td style="text-align:center;"> Congdon (2009) </td> </tr> </tbody> </table> ??? * Models impose demographic regularity: plausible shape * Bayesian Hierarchical Models (BHM): leverage similarities in the data * Bayesian models offer an additional tool to stabilize small-scale mortality estimation * Bayesian modeling: natural framework for __hierachical__ modeling * Admin1 = province and Admin2 = district/municipality: we care about Admin 2 mortality. --- <table class="table" style="width: auto !important; margin-left: auto; margin-right: auto;"> <thead> <tr> <th style="empty-cells: hide;border-bottom:hidden;" colspan="1"></th> <th style="border-bottom:hidden;padding-bottom:0; padding-left:3px;padding-right:3px;text-align: center; " colspan="2"><div style="border-bottom: 1px solid #ddd; padding-bottom: 5px; ">Demographic regularity</div></th> <th style="border-bottom:hidden;padding-bottom:0; padding-left:3px;padding-right:3px;text-align: center; " colspan="4"><div style="border-bottom: 1px solid #ddd; padding-bottom: 5px; ">Pooling</div></th> <th style="empty-cells: hide;border-bottom:hidden;" colspan="1"></th> </tr> <tr> <th style="text-align:center;"> Model name </th> <th style="text-align:center;"> Functional form </th> <th style="text-align:center;"> Penalize deviations </th> <th style="text-align:center;"> Space </th> <th style="text-align:center;"> Age </th> <th style="text-align:center;"> Time </th> <th style="text-align:center;"> Sex </th> <th style="text-align:center;"> References </th> </tr> </thead> <tbody> <tr grouplength="2"><td colspan="8" style="border-bottom: 1px solid;"><strong>Non-BHM</strong></td></tr> <tr> <td style="text-align:center;padding-left: 2em;" indentlevel="1"> D-splines </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> X </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> Schmertmann (2021) </td> </tr> <tr> <td style="text-align:center;padding-left: 2em;" indentlevel="1"> TOPALS </td> <td style="text-align:center;"> X </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> Gonzaga & Schmertmann (2016) </td> </tr> <tr grouplength="4"><td colspan="8" style="border-bottom: 1px solid;"><strong>BHM</strong></td></tr> <tr> <td style="text-align:center;padding-left: 2em;" indentlevel="1"> BHM TOPALS </td> <td style="text-align:center;"> X </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> X </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> X </td> <td style="text-align:center;"> Rau & Schmertmann (2020) </td> </tr> <tr> <td style="text-align:center;padding-left: 2em;" indentlevel="1"> BHM SVD </td> <td style="text-align:center;"> X </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> X </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> X </td> <td style="text-align:center;"> </td> <td style="text-align:center;"> Alexander et al. (2017) </td> </tr> <tr> <td style="text-align:center;padding-left: 2em;text-decoration: underline;" indentlevel="1"> BHM </td> <td style="text-align:center;text-decoration: underline;"> </td> <td style="text-align:center;text-decoration: underline;"> </td> <td style="text-align:center;text-decoration: underline;"> </td> <td style="text-align:center;text-decoration: underline;"> X </td> <td style="text-align:center;text-decoration: underline;"> X </td> <td style="text-align:center;text-decoration: underline;"> X </td> <td style="text-align:center;text-decoration: underline;"> Zhang et al. (2019) </td> </tr> <tr> <td style="text-align:center;padding-left: 2em;text-decoration: line-through;" indentlevel="1"> BREM </td> <td style="text-align:center;text-decoration: line-through;"> </td> <td style="text-align:center;text-decoration: line-through;"> </td> <td style="text-align:center;text-decoration: line-through;"> X </td> <td style="text-align:center;text-decoration: line-through;"> X </td> <td style="text-align:center;text-decoration: line-through;"> </td> <td style="text-align:center;text-decoration: line-through;"> X </td> <td style="text-align:center;text-decoration: line-through;"> Congdon (2009) </td> </tr> </tbody> </table> --- class: inverse, center, middle # Resarch questions: .left[ ## 1. Compare the performance of modelling approaches encountered in small-area mortality estimation applied in a subnational context ## 2. Assess the capability to reliably estimate quantities of interest in a subnational context ] --- # Methodology __Need true subnational mortality age schedules to compare models__
.highlight[Simulation of a fictitious country:] Simulate mortality age schedules of subnational areas by gender over 10 years -- ### Comparison metrics .leftcol70[ * RMSE = `\(\sqrt{\frac{1}{G}\sum^G_{x=1}(\hat{m}_x - m^{sim}_x)^2}\)` * Coverage = `\(\frac{1}{G}\sum^G_{x=1}1[m_x^{sim} \geq l_x]1[m_x^{sim} < r_x]\)` ] .rightcol30[ `\(m_x^{sim}\)` known (simulated) ] <br/> computed across scenarios (see later) ??? * accuracy * calibration --- class: inverse, center, middle # Scenarios --- # Three dimensions * Two administrative levels: .highlight[Districts] & .highlight[Municipalities]
Calibrated on Belgian administrative subdivisions and population quantiles <br/> * Two spatial structures: .highlight[Hierarchy] & .highlight[Random]
Performance of BHM when assumed spatial hierarchy is incorrect <br/> * Two levels of disparity: .highlight[Realistic] & .highlight[High]
Simulated difference in `\(e^0\)` within the country is around 5 and 10 years --- class: inverse, center, middle # Performance comparison --- # Average RMSE <img src="data:image/png;base64,#rmse_new.jpg" width="130%" style="display: block; margin: auto;" /> ??? * BHM SVD and BHM * Districts
Municipalities:
RMSE * BHMs stabilize RMSE at smaller pop sizes * Higher disparity and smaller population sizes:
RMSE --- # Average 95% coverage <img src="data:image/png;base64,#cov_new.jpg" width="130%" style="display: block; margin: auto;" /> ??? * BHM TOPALS over-confident * Disparity impacts coverage at small population sizes --- class: inverse, center, middle # Indicators of interest in a subnational context (with BHM SVD) --- # Life expectancy at birth <div class="figure" style="text-align: center"> <img src="data:image/png;base64,#e0_diag_dist.jpg" alt="Districts, realistic disparity" width="45%" /> <p class="caption">Districts, realistic disparity</p> </div> <div class="figure" style="text-align: center"> <img src="data:image/png;base64,#e0_diag_mun.jpg" alt="Municipalities, realistic disparity" width="45%" /> <p class="caption">Municipalities, realistic disparity</p> </div> * Correlation = 0.96 for both districts and municipalities --- # Ranking according to life expextancy at birth <img src="data:image/png;base64,#e0.jpg" width="90%" style="display: block; margin: auto;" /> --- # Ranking according to life expextancy at birth <div class="figure" style="text-align: center"> <img src="data:image/png;base64,#ranke0.jpg" alt="Districts, realistic disparity" width="90%" /> <p class="caption">Districts, realistic disparity</p> </div> * More than 80% of expected ranks in `\(e^0\)` are off by less than 5 position out of 43 * Ranking municipalities leads to much higher difference * Wider credible intervals for `\(e^0\)` * Higher number of subnational areas --- # Lifespan variation <div class="figure" style="text-align: center"> <img src="data:image/png;base64,#sd.jpg" alt="Districts, realistic disparity" width="90%" /> <p class="caption">Districts, realistic disparity</p> </div> * Correlation = 0.84 --- # Conclusions * Simulation offers an interesting setup to compare models over scenarios * RMSE for districts `\(<\)` RMSE for municipalities * BHM SVD (BHM) has a better performance in terms of average RMSE and coverage than the other models considered * Hierarchical models stabilize their RMSE for smaller population sizes * Performance of models are negatively impacted by higher disparity for smaller population sizes * Incorrectly specifying the spatial hierarchy in the hierarchical models does not significantly affect the performance * BHM SVD allows to reliably estimate demographic indicators at district level (life expectancy, ranking, lifespan variation) but metrics related to the overall distribution within the country are less reliable for municipalities (ranking) --- class: inverse, center, middle # Thank you for your attention ! <br/> <br/> .left[
.link-email[[benjamin-samuel.schluter@uclouvain.be](benjamin-samuel.schluter@uclouvain.be)]
.link-email[[http://benjisamschlu.github.io/DDD/index.html](http://benjisamschlu.github.io/DDD/index.html)]
.link-email[[@benjisamschlu](https://github.com/benjisamschlu)] ] --- class: inverse, center, middle # Additional slides --- # Methodology ### Simulation's requirements * Coherent mortality age schedules * Realistic range of life expectancy at birth within the country * Time dimension * Mortality decreases over time * Temporal stability of the best/worst performing areas * At least two administrative levels ??? * Previous work in France and Germany showed that `\(\Delta e^0 \leq 5-6\)` --- class: inverse, center, middle # Simulation setup --- # Provincial mortality .leftcol65[ <img src="data:image/png;base64,#map_be.jpg" width="80%" style="display: block; margin: auto;" /> ] .rightcol35[ Mortality of 10 provinces .center[=] Mortality of 10 HMD countries in 2003, by gender ] -- .leftcol65[ <img src="data:image/png;base64,#index_files/figure-html/dev_brass-1.png" width="80%" style="display: block; margin: auto;" /> ] .rightcol35[ <br/> .highlight[Brass relational model] `$$logit(l_x^{area}) = a + b \cdot logit(l^{prov.}_x)$$`] ??? * Spatial structure of Belgium for our simulation
Admin1= 10 provinces, admin2= 43 districts or 581 municipalities * 1st step: associate to each province a mortality age schedule from a country in the HMD in 2003 for both male and female * a: level of mortality * b: relationship between young and old mortality --- # Correlation between Brass parameters .leftcol65[ <img src="data:image/png;base64,#index_files/figure-html/corr_brass_pars-1.png" width="80%" style="display: block; margin: auto;" /> ] -- .rightcol35[ <br/> .highlight[Estimate multivariate random walks with drift on HMD country] ] -- .leftcol65[ .highlight[For each area simulate] `$$\begin{bmatrix} a_t^f \\ b_t^f \\ a_t^m \\ b_t^m \end{bmatrix} \sim N( \begin{bmatrix} a_{t-1}^f + \hat{drift}^f \\ b_{t-1}^f + \hat{drift}^f \\ a_{t-1}^m + \hat{drift}^m \\ b_{t-1}^m + \hat{drift}^m \end{bmatrix} , \hat{\boldsymbol\Sigma}^{rescaled})$$` ] .rightcol35[ <br/> * Drift
Temporal improvement (differs by province and gender) * Scaling covariance matrices
Stability over time ] ??? * ... a and b over __10 years__ for both gender and then, used the Brass relational model to obtain its survival curves and hence, mortality age schedules. Repeat that process for each departments --- class: inverse, center, middle # Simulation outputs --- # Simulated life expectancy at birth .pull-left[ <div class="figure" style="text-align: center"> <img src="data:image/png;base64,#index_files/figure-html/e0_real_nuts-1.png" alt="Districts, realistic disparity" width="95%" /> <p class="caption">Districts, realistic disparity</p> </div> ] -- .pull-right[ <div class="figure" style="text-align: center"> <img src="data:image/png;base64,#index_files/figure-html/e0_real_lau-1.png" alt="Municipalities, realistic disparity" width="95%" /> <p class="caption">Municipalities, realistic disparity</p> </div> ] -- .pull-left[ <div class="figure" style="text-align: center"> <img src="data:image/png;base64,#index_files/figure-html/e0_ineq_nuts-1.png" alt="Districts, high disparity" width="95%" /> <p class="caption">Districts, high disparity</p> </div> ] -- .pull-right[ <div class="figure" style="text-align: center"> <img src="data:image/png;base64,#index_files/figure-html/e0_ineq_lau-1.png" alt="Municipalities, high disparity" width="95%" /> <p class="caption">Municipalities, high disparity</p> </div> ] ??? * Realistic mortality decreases over time * Realistic disparity in `\(e^0\)` * Temporal stability in performance